We have the heap lock here, so should not need to use atomic store operations. Atomic operations have a performance penalty that I think we want to avoid.

If we do need to use Atomic operations, would prefer &_region_counts[int(ShenandoahFreeSetPartitionId::Mutator)] notation for the array elements.

Are you now calling this directly from the CAS allocators? So you want to not have to assert heap locked and that is why we make the used accounting atomic?

My preference would be to avoid this need by counting the entire region as used at the time the region becomes directly allocatable.

Probably the preferred approach here is to "pre-retire" regions when they are made directly allocatable. When the region is pre-retired, it is taken out of the partition, so assert_bounds no longer applies to this region.

Prefer not to make these volatile, as that imposes a compiler overhead.

These changes are beyond the scope of planned topic. I think we need to consider them more carefully. Would prefer not to mix the two. (and I personally believe the original implementation has better performance, but feel free to prove me wrong.)

Not clear to me what prefix par_ represents. Parallel allocate (without lock?)

This code has a lot more synchronization overhead than what is required for CAS allocations. load_acquire() forces a memory fence. All writes performed by other threads before the store_release() must be visible to this thread upon return from load_acquire. I would like to see some documentation that describes the coherency model that we assume/require here. Can we write this up as a comment in the header file?

Personal preference: I think there are many situations where we get better performance if we allow ourselves to see slightly old data, and we can argue that the slightly old data is "harmless". For example, if some other thread replaces the directly_allocatable_region[N] while we're attempting to allocate from directly_allocatable_region[N], we might attempt to allocate from the original region and fail. That's harmless. We'll just retry at the next probe point. If multiple probes fail to allocate, we'll take the synchronization lock and everything will be resolved there. The accumulation of atomic volatile access has a big impact on performance. I've measured this in previous experiments. You can do the same.

Would be preferable to allow this thread to retire all ready-to-retire regions in the directly allocatable set (not just the three that I know about) while it holds the heap lock. We do not necessarily need to keep a separate per-thread representation of ready-to-retire shared regions. This is a very rare event. Just iterate through the 13 (or whatever) regions in the directly allocatable set and ask for each whether end-top is less than min plab size.

Out of time to dive deep into this right now. Wonder if it makes sense to randomly generate a hash for each thread and store this into a thread-local field. Might provide "randomness" and locality.